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DETAILED ACTION 
Priority 

1 . Receipt is acknowledged of papers submitted under 35 U.S.C. 1 1 9(a)-(d), which 
papers have been placed of record in the file. 

Specification 

2. 35 U.S.C. 112, first paragraph, requires the specification to be written in "full, 
clear, concise, and exact terms." The specification is replete with terms that are not 
clear, concise and exact. The specification should be revised carefully in order to 
comply with 35 U.S.C. 112, first paragraph. Examples of some unclear, inexact or 
verbose terms used in the specification are: "superposed", "working properties" (pg. 1, 
line 21), "remarkable" (21:6; 22:13, 22:21), etc. 

3. The disclosure is objected to because of the following informalities: pg. 1 , lines 
16 and 24, the word "superposes" is used, where the correct term is "superimposes"; 
pg. 1 , lines 12-15, the sentence is awkwardly constructed and hard to read, where the 
correct phrasing would be "Recently, an apparatus has been actively proposed that 
would use mixed reality techniques..."; line 20, the terms "in this apparatus to improve" 
where the correct wording would be "that this apparatus will improve"; pg. 21 , line 1 7, 
the terms "operation with use of is used and is redundant and confusing - proper 
wording would be "operation of; etc. 

The above examples serve to illustrate the kinds of errors that the specification is 
replete with. Appropriate correction is required. 
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4. A substitute specification in proper idiomatic English and in compliance with 37 
CFR 1.52(a) and (b) is required. The substitute specification filed must be accompanied 
by a statement that it contains no new matter. 

5. The lengthy specification has not been checked to the extent necessary to 
determine the presence of all possible minor errors. Applicant's cooperation is 
requested in correcting any errors of which applicant may become aware in the 
specification. 

Drawings 

6. The drawings are objected to because an element in Figs. 4-6 has the 
description "Target that user pays attention." This description is inaccurate and 
incomplete, and should read "Target that user pays attention to" or "Target that has 
user's attention" or "Target that user notices." Corrected drawing sheets in compliance 
with 37 CFR 1 .121(d) are required in reply to the Office action to avoid abandonment of 
the application. Any amended replacement drawing sheet should include all of the 
figures appearing on the immediate prior version of the sheet, even if only one figure is 
being amended. The figure or figure number of an amended drawing should not be 
labeled as "amended." If a drawing figure is to be canceled, the appropriate figure must 
be removed from the replacement sheet, and where necessary, the remaining figures 
must be renumbered and appropriate changes made to the brief description of the 
several views of the drawings for consistency. Additional replacement sheets may be' 
necessary to show the renumbering of the remaining figures. The replacement sheet(s) 
should be labeled "Replacement Sheet" in the page header (as per 37 CFR 1 .84(c)) so 
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as not to obstruct any portion of the drawing figures. If the examiner does not accept the 
changes, the applicant will be notified and informed of any required corrective action in 
the next Office action. The objection to the drawings will not be held in abeyance. 

Claim Rejections - 35 USC § 101 

7. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

8. Claims 1-29 are rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. 

Claim 29 recites a "program" that performs the method, but is not limited to being 
performed by or on a computer or on computer-readable media. As such, it recites 
functional descriptive material (computer program / software per se) and is non- 
statutory. 

Claims 22-28, the method claims are thusly non-statutory because one of the 
implementations is non-statutory. See MPEP 2106, IV-A. 

Claims 1-21 are non-statutory because they read on software. Examiner 
emphasizes for claim 1 + that each of the elements in the body of the claim includes the 
word unit which when combined with page 26 (lines 3-10) of the spec, renders the claim 
broad enough to read on a software implementation. Examiner recommends either 
changing the word "unit" in the claims or editing the specification. Amendments may not 
introduce new matter. 

To expedite a complete examination of the instant application the claims rejected 
under 35 U.S.C. 101 (nonstatutory) above are further rejected as set forth below in 
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anticipation of applicant amending these claims to place them within the four statutory 
categories of invention. 

Claim Objections 

9. Claim 2 is objected to because of the following informalities: the terms "of each 
user's viewpoint" are used, where the appropriate terminology would be "for each user's 
viewpoint". 

10. Claim 4 is objected to because of the following informalities: the terms "on the 
virtual world" are used, where the correct terminology would be "in the virtual world." 

1 1 . Claim 1 1 is objected to because of the following informalities: the term "said the" 
is used, where the correct usage would be "the said". 

12. Claim 1 1 is objected to because of the following informalities: the term 
"superpose" is used, where the correct term is "superimposed". 

13. Claims 13-16 and 19 are objected to because of the following informalities: the 
terms "target that the user pays attention" are used, wherein the correct wording would 
be "Target that user pays attention to" or "Target that has user's attention" or "Target 
that user notices" or the terminology used in later claims - "attention target". 

14. Claim 20 is objected to because of the following informalities: the terms "a 
direction of other user" are used, wherein the correct wording should be "a direction of 
another user" or "another user's direction". 

15. Claim 21 is objected to because of the following informalities: the terms "a 
position of other user" are used, wherein the correct wording should be "a position of 
another user" or "another user's position." 
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Appropriate correction is required 

Claim Rejections - 35 USC §112 

16. The following is a quotation of the second paragraph of 35 U.S.C. 1 1 2: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

17. Claims 2, 4-6, 9, and 24-26 are rejected under 35 U.S.C. 112, second paragraph, 
as being indefinite for failing to particularly point out and distinctly claim the subject 
matter which applicant regards as the invention. 

Claim 2 recites the limitation "said information presentation apparatus" in lines 3- 
4. There is "an information presentation apparatus" and there are "plural information 
presentation apparatuses." As such, it is unclear which information presentation 
apparatus applicant is referring to. 

Claims 4-5 and 9 are rejected because the term "discrimination information" 
(lines 22-23 for claim 4, lines 6-7 for claim 5, etc.) lacks adequate definition in the art, 
and the nature of the information being conveyed is unknown, given that user pose and 
position data are already obtained from claim 1 and later in the claim applicant recites 
"position and pose information indicating the arrangement of the CG object". As such, it 
is unknown what information would be conveyed. 

Regarding claim 6, the phrase "and the like" renders the claim(s) indefinite 
because the claim(s) include(s) elements not actually disclosed (those encompassed by 
"and the like"), thereby rendering the scope of the claim(s) unascertainable. See MPEP 
§ 2173.05(d). 
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Claims 24-26 are rejected because the terms "synthesized to the synthesized 
image" render the claim indefinite because the language is redundant and it is unclear 
how such an annotation is synthesized to the image that has already been synthesized 
- e.g. a merge operation, a overlay operation, etc. 

Claim Rejections - 35 USC § 102 

18. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

19. Claims 1, 2, and 9 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Thomas (Thomas et al. "First Person Indoor / Outdoor Augmented Reality Application: 
ARQuake"-see MPEP 201.13, subsection lll)(Thomas'). 

As to claim 1 , 

An information presentation apparatus comprising: 
-User operation input unit, adapted to input an operation of a user (Thomas p. 78, 
section 3.1, "Haptic gun"); 

-User viewpoint position and pose measurement unit; adapted to measure a position 
and pose at a user's viewpoint (section 2, pg. 76, particularly section 2.2, pg. 77 where 
the specific tracking mechanisms are disclosed (and a head orientation measurement 
unit); further the fiducial markers (e.g. Fig. 9, pg. 82) serve to correct registration areas 
in and close to buildings); 
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-Model data storage unit, adapted to store virtual world model data, real world model 
data, and data necessary to generate a virtual world image (section 2.2 - the hard drive 
in the notebook computer (hard drive is inherent in notebook PC); Quake inherently 
stores virtual world data and data necessary to generate a virtual world; PC has video 
input put into it for fiducial mark recognition - see discussion in background (e.g. section 
2) on how the software libraries required a camera. Although camera is not explicitly 
mentioned, it is obviously inherent.); 

-Annotation data storage unit, adapted to store data necessary to be added to a real 
world and a virtual world and then displayed (section 2.2 - the hard drive in the 
notebook computer (hard drive is inherent in notebook PC); as shown in Figs. 6 and 7, 
data from the virtual world is displayed on the HUD, which shows data annotated by the 
game.); 

-Virtual image generation unit, adapted to generate an image of the virtual world by 
using information in said user viewpoint position and pose measurement unit, said 
model data storage unit and said annotation data storage unit (section 2.2 - the 
notebook PC (the graphics card would provide this functionality); 
-User viewpoint image input unit, adapted to capture an image of the real world viewed 
from the user's viewpoint (the user display in section 2.2 is transparent, and there is an 
inherent camera as discussed above (so that the fiducials could be recognized)); and 
-Image display unit, adapted to display an image obtained by synthesizing the image 
generated by said virtual image generation unit and the image obtained by said user 
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viewpoint image input unit, on an image display device of the user (heads-up display for 
user as disclosed in section 2.2 and in Figs. 6 and 7, for example). 
As to claim 2, 

An information presentation apparatus according to claim 1 , wherein plural 
information presentation apparatuses are provided, and said information presentation 
apparatus is connected to other information presentation apparatus through a 
transmission channel so as to exchange communication data (section 5.1 - multi- 
player. Two players are connected wirelessly using the WaveLAN network. Multiple 
terminals / wearable computers clearly represent 'plural information presentation 
devices/ Also, each platform has multiple Information presentation devices' in the 
haptic gun that presents sensory information as well as the HUD. In section 5.2, 
multiple players are discussed.) 

As to claim 9, 

An information presentation apparatus according to claim 1 , wherein said virtual image 
generation unit draws the information stored in said model data storage unit from the 
user's viewpoint in computer graphics to generate the image of the virtual world viewed 
from the user's viewpoint, by using the position and pose information at the user's 
viewpoint obtained from said user viewpoint position and pose measurement unit. 
(Figs. 6 and 7 are views of this phenomenon. The model of the virtual world is hosted 
on the notebook computer (inherent in the Quake software game) and clearly the 
positions of monsters, etc., are drawn with respect to the player based on the 
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orientation. Further, position and pose information are known based on 6DOF 
measurement setup that is on the mobile computing platform (see section 2)). 

Claim Rejections - 35 USC § 103 

20. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

21 . Claims 1-3 and 7-10 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Thomas (Thomas et al. "ARQuake: An Indoor/Outdoor Augmented Reality First 
Person Application." (2000))(T2000') in view of Piekarski (Piekarski et al. "Integrating 
virtual and augmented realities in an outdoor application".) 

An information presentation apparatus comprising: 
-User operation input unit, adapted to input an operation of a user (T2000 - "two button 
input device" (section 3, pg. 141))(Piekarski Fig. 3, the Phoenix forearm keyboard); 
-User viewpoint position and pose measurement unit, adapted to measure a position 
and pose at a user's viewpoint (T2000 section 2.3, where the specific tracking 
mechanisms are disclosed (and a head orientation measurement unit); further the 
fiducial markers (T2000, Fig. 3, pg. 143) serve to correct registration areas in and close 
to buildings)(similar mechanisms in Piekarski); 

-Model data storage unit, adapted to store virtual world model data, real world model 
data, and data necessary to generate a virtual world image (T2000 section 2.3 - the 
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hard drive in the notebook computer (hard drive is inherent in notebook PC); Quake 
inherently stores virtual world data and data necessary to generate a virtual world); 
-Annotation data storage unit, adapted to store data necessary to be added to a real 
world and a virtual world and then displayed (Piekarski section 2.3 - the hard drive in 
the notebook computer (hard drive is inherent in notebook PC); as shown in Figs. 6, 7, 
and 8b data from the virtual world is displayed on the HUD - that is the augmented 
view, which shows data annotated by the game, and annotated overlay information is 
shown. In Fig. 7, range information and markers are shown as well.); 
-Virtual image generation unit, adapted to generate an image of the virtual world by 
using information in said user viewpoint position and pose measurement unit, said 
model data storage unit and said annotation data storage unit (Piekarski section 3.1 - 
the notebook PC (the graphics card would provide this functionality) - inherent in 
notebook computer. )(T2000 2.3 again - that is, the notebook computer would supply 
this functionality); 

-User viewpoint image input unit, adapted to capture an image of the real world viewed 
from the user's viewpoint (T2000, section 7, a camera for use with the system is 
disclosed. Also, given that the fiducials could be recognized, discussion in sections 4.2 
and 4.3, and fiducials shown in Fig. 3, a camera is inherent); and 
-Image display unit, adapted to display an image obtained by synthesizing the image 
generated by said virtual image generation unit and the image obtained by said user 
viewpoint image input unit, on an image display device of the user (T2000 heads-up 
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display for user as disclosed in section 2.3, overlay view and a sample view is shown 
Fig. 2, for example, with virtual world information displayed). 

Reference T2000 teaches all the limitations of the claim except explicit use of 
annotation database (but that is fairly obvious in the presence of the extra data from the 
game shown in the overlay in Fig. 2). Reference Piekarski discloses the use of 
annotation and overlay techniques clearly in Figs. 6 and 7 that show the augmented 
reality view with annotated objects and their information. It would have been obvious to 
one having ordinary skill in the art at the time the invention was made to combine the 
augmented reality system with camera of T2000 with the annotated augmented reality 
system of Piekarski, since the non-camera system of Piekarski could achieve an 
accuracy of 5 meters in most cases, but the incorporation of the fiducials via the camera 
in the later version of the system improved accuracy to 10cm indoors (and GPS is said 
to not function indoors - see T2000, section 4 for discussion). 

As to claim 2, 

An information presentation apparatus according to claim 1 , wherein plural information 
presentation apparatuses are provided, and said information presentation apparatus is 
connected to other information presentation apparatus through a transmission channel 
so as to exchange communication data (Piekarski, last paragraph of section 5.2, 
multiple users are disclosed all having information presentation apparatuses ('wearable 
computers') and they are connected over a wireless LAN)(T2000 teaches wireless 
transmission of . 
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Reference T2000 does not explicitly teach the use of multiple information 
presentation apparatuses on wireless channels. Reference Piekarski teaches the use 
of multiple users in a simulation environment connected through a wireless LAN [Note, 
however, that T2000 does use a wireless LAN connection, as it uses the Tinmith-2 
platform, which inherently has this component, as described in earlier papers by this 
group]. It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality of T2000 with the wireless 
connectivity and multiple information presentation apparatuses of Piekarski, as this 
enables the augmented reality to be expanded for use in multi-user environments with 
collaboration (section 6 Piekarski). 

As to claim 3, 

An information presentation apparatus according to claim 2, wherein the communication 
data includes an identification number of each user using said information presentation 
apparatus, a name for discriminating each user, position and pose information of each 
user's viewpoint, operation information of each user, and annotation data. 

Reference T2000 does not explicitly teach these limitations. Reference Piekarski 
teaches the use of multiple users and as well as simulated objects (planes, ships, tanks, 
etc.). Each object/unit, as shown in the overlays in Figs. 6, 7, and 8B has a label with a 
number (e.g. CHOPPER0, etc.) where this could obviously be split to be CHOPPER 1 , 
with the name and number separate, thus meeting that recited limitation. It is prima 
facie obvious that the same information would be provided on other users in the system. 
Further, each object in Fig. 7 is labeled with a distance from the user and the overall 
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position, and the angular location (e.g. on 360 degree scale around the user) is 
specified for some objects, so it could obviously be specified for all of them. Finally, 
each user obviously has position and pose information associated with them, and it 
would be obvious to display that (although speed, location, and direction information on 
the overlay would convey the same resultant information). It would be obvious to add 
extra information to specific users versus virtual entities to cover "operation information" 
and "annotation data". Further, the system very clearly (see section 2.2) allows users to 
add entities and relative position information, as well as information on the "nature" of 
the entity being added. Since the users have a forearm keyboard, that would allow the 
addition of annotation information. It would have been obvious to one having ordinary 
skill in the art at the time the invention was made to combine the augmented reality of 
T2000 with the labeling and annotation of Piekarski, as this enables the augmented 
reality to be effective in multi-user environments with collaboration and convey much 
more information visually (sections 6 and 2.2 respectively, Piekarski). 
As to claim 7, 

An information presentation apparatus according to claim 1 , wherein said annotation 
data storage unit can store annotation data being additional information to be displayed 
on the real world and the virtual world. 

Reference T2000 does not specifically teach annotation data in this format (e.g. 
Quake will display player health, armor, etc., overlaid on the game screen (see Fig. 2 
T2000). Reference Piekarski clearly shows in Figs. 6, 7, and 8B the use of annotated 
data. Further, the system very clearly (see section 2.2) allows users to add entities and 
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relative position information, as well as information on the "nature" of the entity being 
added. Since the users have a forearm keyboard, that would allow the addition of 
annotation information. It would have been obvious to one having ordinary skill in the 
art at the time the invention was made to combine the augmented reality of T2000 with 
the annotation of Piekarski, as this would allow the user to obtain information on the 
nature of entities (e.g. enemy units) and transmit the nature of new ones spotted to 
other soldiers / collaborators, etc. (see 2.2 and 6 in Piekarski). 
As to claim 8, 

An information presentation apparatus according to claim 7, wherein the annotation 
data includes position and pose information of an object arranged on the real world and 
the virtual world, discrimination information of the object, and text, symbol and image 
information for indicating information of the object to the user. 

Reference T2000 does not specifically teach annotation data in this format (e.g. 
Quake will display player health, armor, etc., overlaid on the game screen (see Fig. 2 
T2000). Reference Piekarski clearly shows in Figs. 6, 7, and 8B the use of annotated 
data. Further, the system very clearly (see section 2.2) allows users to add entities and 
relative position information, as well as information on the "nature" of the entity being 
added. Piekarski clearly shows the position, speed, etc., of other entities in the 
environment in the overlay view (see Figs. 6, 7, and 8B), and as discussed above, 
would obviously convey that information (and logically any other required to convey 
pose / position information) about other users in the system, and this would include 
discrimination information. As shown in Figure 7, the objects have names and 
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numbers; obviously, these could prima facie be separated so as to provide text. The 
symbol provided could be the arrows of Figs. 6, 7, or 8B, and the image would be, for 
example, the avatar shown in T2000 (Fig. 2) that shows the player. It would have been 
obvious to one having ordinary skill in the art at the time the invention was made to 
combine the augmented reality of T2000 with the annotation of Piekarski, as this would 
allow the user to obtain information on the nature of entities (e.g. enemy units) and 
transmit the nature of new ones spotted to other soldiers / collaborators, etc. (see 2.2 
and 6 in Piekarski). 
As to claim 9, 

An information presentation apparatus according to claim 1 , wherein said virtual image 
generation unit draws the information stored in said model data storage unit from the 
user's viewpoint in computer graphics to generate the image of the virtual world viewed 
from the user's viewpoint, by using the position and pose information at the user's 
viewpoint obtained from said user viewpoint position and pose measurement unit. 

In T2000, Fig. 2 is a view of this phenomenon. The model of the virtual world is 
hosted on the notebook computer (inherent in the Quake software game) and clearly 
the positions of monsters, etc., are drawn with respect to the player based on the 
orientation. Further/ position and pose information are known based on 6DOF 
measurement setup that is on the mobile computing platform (see section 2.3)). The 
monsters and objects projected through the HUD are drawn with respect to the user 
position and pose as recited; that is prima facie obvious. Since only the primary 
reference is utilized, no separate motivation or combination is required. 
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As to claim 10, 

An information presentation apparatus according to claim 9, wherein said virtual image 
generation unit has a function to transfer data to a transmission channel and a function 
to receive data from the transmission channel. 

Reference T2000 does not explicitly teach this limitation (though it is inherent in 
the Tinmith wearable computer platform, as earlier work by Thomas has shown). 
Reference Piekarski, utilizing the Tinmith system, explicitly teaches this limitation - 
wireless data channels for the wearable computers - in section 4. It would have been 
obvious to one having ordinary skill in the art at the time the invention was made to 
combine the augmented reality system of T2000 and the wireless communication 
channel of Piekarski, as this would allow the user to obtain information on the nature of 
entities (e.g. enemy units) and transmit the nature of new ones spotted to other soldiers 
/ collaborators, etc. (see 2.2 and 6 in Piekarski), an advantage over the (assumed) 
single-user system in T2000. 

22. Claims 4 and 5 are rejected under 35 U.S.C. 103(a) as unpatentable over T2000 
in view of Piekarski as applied to claim 1 above, further in view of Piekarski (Piekarski et 
al. "Tinmith-Metro: New Outdoor Techniques for Creating City Models with an 
Augmented Reality Wearable Computer. ,, )( t P200r). 
As to claim 4, 

An information presentation apparatus according to claim 1 , wherein the virtual world 
model data includes three-dimensional coordinates of vertices of a polygon of a virtual 
computer graphics (CG) object arranged on the virtual world, structure information of 
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faces of the polygon, discrimination information of the CG object, color information, 
texture information, a size of the CG object, and position and pose information 
indicating the arrangement of the CG object on the virtual world. 

References T2000 and Piekarski do not explicitly teach all the limitations of the 
claims. Reference Piekarski does teach the overlay of a CG object with position and 
pose information (e.g. the helicopter shown in Fig. 8C would show up on the user's 
display, and be overlaid with the information shown in Fig. 7 / 8B that shows distance 
and name of the object - it would be an obvious variation to show all data at once), but 
does not teach all the information about structures of the polygons, etc. 

Reference P2001 - a more advanced version of the Tinmith platform used in 
T2000 and Piekarski - teaches the use of the system in designing building for urban 
environments, with the accordant information about polygons etc. As shown in P2001 
Fig. 2A-2C, different versions of the same building, made of polygons, are shown. Each 
polygon is constructed from lines and planes (Fig. 2A) and then the details of each 
polygon (face, etc.) are shown in Fig. 2B (in the left margin of the image, specific 
information on the faces is shown). Fig. 2C shows overall orientation of the building 
with more specific attributes shown. As shown in P2001 Fig. 2B, each face is shown 
with the texture and color applied, with that information also supplied in the left margin, 
since the faces are being constructed and illustrated during the design process. P2001 
teaches in section 3 different operations that are used to construct buildings and objects 
in 2-D and then extend them to 3-D or simply form them in 3-D. On section 7, pg. 37, 
P2001 teaches that the Tinmith platform can render the scenes into primitive triangle 
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lists or VRML files. As one of ordinary skill in the art is doubtless aware, Virtual Reality 
Markup Files contain all the vertices of objects described therein so that the objects can 
be accurately rendered. Given that, and Fig. 2B, it would be obvious to show the 
specific coordinates of the vertices as argued by applicant. Fig 2C represents final 
coordinate information of the object in the real world - also Fig. 3, which coupled with 
the coordinate information shown in Fig. 2B of the faces and the position and pose 
information revealed in Piekarski, would make it prima facie obvious to have the pose 
and position information of the structure shown, as well as any additional information 
required. The specific information in Figs. 2A and 2B during the construction process 
show size of the various faces of the polygons / buildings, etc. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality systems of T2000 and Piekarski 
with the construction systems of P2001, since system of P2001 allows users to create 
objects for use in the augmented reality worlds of T2000 and Piekarski (common 
technology platform)(see sections 3 and 4 for specific construction methods). 

As to claim 5, 

An information presentation apparatus according to claim 1 , wherein the real world 
model data includes three-dimensional coordinates of vertices of a polygon of an object 
existing in the real world merged with the virtual world, structure information of faces of 
the polygon, discrimination information of the object, a size of the object, and position 
and pose information indicating the arrangement of the object. 
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See the rejection above for claim 4. The only additional limitation here is the 
limitation that such real world model data includes three-dimensional coordinates, etc. - 
the same limitations as for claim 4. They are all addressed above - and the augmented 
reality system shown in Piekarski has such elements merged anyway, as the helicopter 
(Fig. 8A and 8C) is present in the system as overlaid. Particularly in Piekarski, since no 
camera is present, the real world model must prima facie include the virtual world data 
in order to generate the virtual camera views discussed therein (see section 2.2). Also, 
in T2000 the Quake monsters are displayed "in spatial context with the physical world" 
which indicates that the data does exist in the real world model as recited above in 
claim 5. Thusly, it would be obvious to use the augmented reality of P2001 with the real 
world model data, as the application is for modeling objects in the real world, and all 
three references utilize the same Tinmith technology platform. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality systems of T2000 and Piekarski 
with the construction systems of P2001 , since system of P2001 allows users to create 
objects for use in the augmented reality worlds of T2000 and Piekarski (common 
technology platform)(see sections 3 and 4 for specific construction methods). 
24. Claim 6 is rejected under 35 U.S.C. 103(a) as unpatentable over T2000 in view 
of Piekarski as applied to claim 1 above, and further in view of Matsumoto (US PGPub 
2002/01 64066)('Matsumoto'). 

As to claim 6, 
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An information presentation apparatus according to claim 1 , wherein the data necessary 
to generate the virtual world image includes internal parameters such as size and angle 
of an image pickup element of an image pickup device of said user viewpoint image 
input unit, an angle of view of a lens, a lens distortion parameter and the like. 

References T2000 and Piekarski do not explicitly teach the inclusion of such 
internal parameters, but it is well known in the art that such parameters are used in the 
calibration of cameras for ranging applications, etc. Specifically, reference T2000 
teaches that the ARToolkit used on the Tinmith platform for image processing contains 
"calibration code for optical and see-through applications." Reference Matsumoto 
teaches the use of these internal parameters for calibration of a camera for three- 
dimensional applications (01 10), specifically view angle information, lens distortion, 
size, ratio of focal length, etc. are determined. The angle of the image pickup unit and 
its location would be known from ranging through GPS / INS, fiducial marks, etc, as well 
known in the art (absolute position - see T2000 for 10cm accuracy indoors, etc.). It 
would have been obvious to one having ordinary skill in the art at the time the invention 
was made to combine the camera / image pickup unit calibration techniques of 
Matsumoto with the augmented reality of T2000 and Piekarski, as T2000 teaches 
calibration code for cameras and Matsumoto enumerates various components that such 
code would perform. 

25. Claims 11-12, 15-18, 22-23, and 27 are rejected under 35 U.S.C. 103(a) as 
unpatentable over T2000 in view of Piekarski as applied to claim 9 above, and further in 
view of Thomas. 
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As to claim 1 1 , 

An information presentation apparatus according to claim 9, wherein said the virtual 
image generation unit has a function to generate an annotation by selecting the 
information to be presented to the user from the annotation data stored in said 
annotation data storage unit on the basis of the position and pose at the user's 
viewpoint obtained from said user viewpoint position and pose measurement unit and 
the position and pose of other user's viewpoint obtained through the transmission 
channel, and to superpose the generated annotation on the image of the virtual world. 

References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Thomas teaches the use in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow 
is taught to mimic the behavior of the player virtual avatar, e.g. it points in the direction 
the user is looking and is noticeable by the other players. Thomas suggests that the 
user would be able to control visibility and location of the pointer. In section 2.2 Thomas 
discloses the use of a sophisticated head-tracking system for pose information. The 
arrow would prima facie follow the user's head orientation. The annotations shown in 
Piekarski, e.g. the overlaid speed, angle, and / or direction indicators, are associated 
with different objects. Interpreting this claim as broadly as possible, the annotations 
shown on screen would clearly depend on where the users are looking, e.g. when the 
arrow indicating objects being pointed to is shown to other users, it is disclosed that in 
this embodiment voice communications are utilized to describe the specific object and 
/or transmit additional information about it. However, the system in Piekarski clearly 
allows users to enter information using the forearm keyboard shown in Fig. 3, and 
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transmits such data to other users and the system generally (as discussed in earlier 
rejections). Therefore, it would logical that the user could text-annotate the arrows, 
particularly if one user were based at a workstation, as is disclosed by Thomas (section 
5.1). Although voice communications are mentioned, data communications channels 
(e.g. the chat features of Quake (inherent to the software)) would be much easier to use 
for annotation purposes, etc. With a forearm keyboard, written annotation in addition to 
commands could be transmitted and seen by other users. It would have been obvious 
to one having ordinary skill in the art at the time the invention was made to combine the 
augmented reality systems of T2000 and Piekarski with the pointers of Thomas, since 
the use of arrows would allow users to communicate - visually - areas where another 
user wanted them to look to convey information, which would be very helpful in a multi- 
user game or simulation (as in Piekarski). 
As to claim 12, 

An information presentation apparatus according to claim 1 1 , wherein the annotation 
includes a symbol, a character string, and image information. 

Reference T2000 does not teach this limitation. Reference Piekarski teaches 
some of the limitations, while reference Thomas teaches the rest. See the above claim 
11 rejection. 

Piekarski teaches in Figure 7 that the objects have names and numbers; 
obviously, these could prima facie be separated so as to provide text. The symbol 
provided could be the arrows of Figs. 6, 7, or 8B, and the image would be, for example, 
the avatar shown in T2000 (Fig. 2) that shows the player. Reference Piekarski teaches 
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the use of multiple users and as well as simulated objects (planes, ships, tanks, etc.). 
Each object/unit, as shown in the overlays in Figs. 6, 7, and 8B has a label with a 
number (e.g. CHOPPER0, etc.) where this could obviously be split to be CHOPPER 1 , 
with the name and number separate, thus meeting that recited limitation. It would have 
been obvious to one having ordinary skill in the art at the time the invention was made 
to combine the augmented reality of T2000 and the annotation of Piekarski with the 
arrows of Thomas, as this would allow the user to obtain information on the nature of 
entities (e.g. enemy units) and transmit the nature of new ones spotted to other soldiers 
/ collaborators, etc. (see 2.2 and 6 in Piekarski) and other players (Thomas). 
As to claim 1 5, 

An information presentation apparatus according to claim 9, wherein said virtual 
image generation unit has a function, in a case where a target that other one or more 
users pay attention is outside a visual range of the user, to generate an annotation 
indicating a direction of the target. 

References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
have a range and direction associated with them. Reference Thomas teaches the use 
in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
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sophisticated head-tracking system for pose information. The arrow would prima facie 
follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 1 1 on that same page). 
It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas, since Piekarski discloses tracking 
arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and target annotation 
in Fig. 7 and Thomas would allow other users / soldiers / players (in the case of T2000) 
to see the targets of interest and guide their attention to them when the targets were 
outside their field of view. 
As to claim 16, 

An information presentation apparatus according to claim 9, wherein said virtual image 
generation unit has a function, in a case where a target that other one or more users 
pay attention is inside a visual range of the user, to generate an annotation indicating 
information of the target. 

See the rejection for claim 15, as this claim is a substantial duplicate - the only 
difference is that the words "inside a visual range" have been substituted for "outside a 
visual range." It would prima facie obvious have the arrow of Thomas point at objects of 
interest to the user, as Thomas is silent on field-of-view issues, meaning that it would be 
obvious to have the arrow pointing at targets of interests to the user, including when the 
target was inside the field of view of another user, as Thomas teaches that the arrow / 
target indicator can be pointed at the specific target by the user. As noted in the 
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rejection for claim 15, Piekarski teaches notation for targets shown on the overlay 
anyway. It would have been obvious to one having ordinary skill in the art at the time 
the invention was made to combine the combine the augmented reality of T2000 and 
the annotation of Piekarski with the arrows of Thomas, since Piekarski discloses 
tracking arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and provide 
target annotation data as shown in Fig. 7 and Thomas would allow other users / soldiers 
/ players (in the case of T2000) to see the targets of interest and guide their attention to 
them when the targets were inside their field of view. 
As to claim 17, 

An information presentation apparatus according to claim 9, wherein said virtual image 
generation unit has a function to generate an annotation of which the attributes of a 
color, a shape and a character type have been changed in regard to each user, and an 
annotation indicating a name for discriminating the user. 

References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
have a range and direction associated with them. Reference Thomas teaches the use 
in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
sophisticated head-tracking system for pose information. The arrow would prima facie 
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follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 11 on that same page). 

Further, it would be obvious to one of ordinary skill in the art that labeling multiple 
targets with the same color markers and annotations would be confusing and create 
visual clutter (it is well known in the human-computer interface art (HCI) that visual 
clutter makes it more difficult to perform depth processing, e.g. it makes the user 
inefficient.) Given this, it would be obvious to make the markers and annotations 
different colors for different targets so that they could be more easily picked out / 
discriminated when there are multiple targets or users visible to the user. This is 
standard procedure in video games. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas, since Piekarski discloses tracking 
arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and provide target 
annotation data as shown in Fig. 7 and Thomas would allow other users / soldiers / 
players (in the case of T2000) to more easily determine their location and pick out their 
actions. 

As to claim 18, 

An information presentation apparatus according to claim 9, wherein said virtual image 
generation unit has a function capable of controlling a generated annotation, by the 
user's operation input to said user operation input unit. 

The system in Piekarski clearly allows users to enter information using the 
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forearm keyboard shown in Fig. 3, and transmits such data to other users and the 
system generally (as discussed in earlier rejections). Therefore, it would logical that the 
user could text-annotate the arrows, particularly if one user were based at a 
workstation, as Thomas discloses in section 5.1. Although voice communications are 
mentioned, data communications channels (e.g. the chat features of Quake (inherent to 
the software)) would be much easier to use for annotation purposes, etc. With a 
forearm keyboard, written annotation in addition to commands could be transmitted and 
seen by other users. 

Reference Thomas teaches the use in section 5 and in Fig. 1 1 of a 3-D pointing 
arrow. This arrow is taught to mimic the behavior of the player virtual avatar, e.g. it 
points in the direction the user is looking and is noticeable by the other players. 
Thomas suggests is that the user would be able to control visibility and location of the 
pointer. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality systems of T2000 and Piekarski 
with the pointers of Thomas, since the use of controllable annotation would allow more 
effective conveyance of information on the nature of a target, and allowing users to 
control the annotation would allow them to communicate — visually, which would be very 
helpful in a multi-user game or simulation (as in Piekarski). 

As to claims 22 and 27 [claim 27 is a software implementation of method claim 
27, and the limitation of software is taught by T2000 and Thomas inherently in the use 
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of the software on the computer, e.g. the Quake program, the use of the ARToolkit 
library, etc.], 

An information processing method comprising the steps of: 

-Inputting viewpoint information of a user (T2000, section 7, a camera for use with the 
system is disclosed. Also, given that the fiducials could be recognized, discussion in 
sections 4.2 and 4.3, and fiducials shown in Fig. 3, a camera is inherent) (T2000 section 
2.3, where the specific tracking mechanisms are disclosed (and a head orientation 
measurement unit); further the fiducial markers (T2000, Fig. 3, pg. 143) serve to correct 
registration areas in and close to buildings)(similar mechanisms in Piekarski); 
-Generating a virtual world image according to the viewpoint information, by using 
previously held virtual world data (T2000 section 2.3 - the hard drive in the notebook 
computer (hard drive is inherent in notebook PC); Quake inherently stores virtual world 
data and data necessary to generate a virtual world; Quake performs this limitation); 
-Generating an annotation concerning an attention target (Thomas teaches the use in 
section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests that the user would be able to 
control visibility and location of the pointer.); and 

-Generating an image obtained by synthesizing an image of a real world, generated 
virtual world image and the generated annotation. (Piekarski section 3.1 - the notebook 
PC (the graphics card would provide this functionality) - inherent in notebook 
computer. )(T2000 2.3 again - that is, the notebook computer would supply this 
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functionality) (T2000 heads-up display for user as disclosed in section 2.3, overlay view 
and a sample view is shown Fig. 2, for example, with virtual world information 
displayed). 

As discussed above, the various embodiments of the Tinmith platform presented 
in the above papers perform various aspects of this functionality. Piekarski clearly 
illustrates the annotation aspect and Thomas clearly establishes the attention target and 
arrow aspect. Very clearly, the user could indicate which target held their attention by 
shooting it with the gun or indicating it with the moving cursor of Thomas (both are 
common techniques in video game systems and well known in the art). See the 
rejection for claim 1 for the specific details of some of the systems, aside from the 
above noted reference locations for specific points in the claim. As discussed in the 
rejection to claim 18, the user could text-annotate or further mark the arrows to indicate 
attention targets. Further, the Quake program is known to construct the virtual world 
from one previously stored on disk (it is inherent). Finally, it is prima facie obvious that 
all the views are merged so that they are displayed in the HUD of the Tinmith system. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality systems of T2000 and Piekarski 
with the pointers of Thomas, since the use of controllable annotation would allow more 
effective conveyance of information on the nature of a target, and allowing users to 
control the annotation would allow them to communicate - visually, which would be very 
helpful in a multi-user game or simulation (as in Piekarski). 

As to claim 23, 
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An information processing method according to claim 22, wherein, in a case where the 
attention target exists outside the synthesized image, an annotation indicating a 
direction of the attention target is generated and synthesized to the synthesized image. 
[Note that in the discussion for the rejection for this claim that follows below, 
"synthesized image" is equivalent to the virtual world image projected onto the HUD, 
e.g. the artificial reality environment, and this is consistent with common use in the art.] 
References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
have a range and direction associated with them. Reference Thomas teaches the use 
in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
sophisticated head-tracking system for pose information. The arrow would prima facie 
follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 1 1 on that same page). 
It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas, since Piekarski discloses tracking 
arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and target annotation 
in Fig. 7 and Thomas would allow other users / soldiers / players (in the case of T2000) 
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to see the targets of interest and guide their attention to them when the targets were 
outside their field of view. 

An information processing method according to claim 22, wherein, in a case 
where the attention target exists inside the synthesized image, an annotation indicating 
additional information for the attention target and having an attribute different from that 
of other annotation is generated and synthesized to the synthesized image. 
26. Claims 13 and 14 are rejected under 35 U.S.C. 103(a) as unpatentable over 
T2000 in view of Piekarski as applied to claim 9 above, and further in view of Pham et al 
(US e^^H^hem'). 

As to claim 1 3, 

An information presentation apparatus according to claim 9, wherein said virtual image 
generation unit has a function to automatically recognize a target that the user pays 
attention. 

References T2000 and Piekarski do not explicitly teach the limitations of the 
claim. However, T2000 teaches the use of fiducials for distance and registration 
purposes for indoors and outdoors applications, which means that the camera and 
computer system are performing image processing and automatic target recognition 
(ATR) of the fiducial marks. Further, Piekarski teaches the use of battlefield simulators 
for military purposes, where automatic target recognition is a fundamental of the art 
(e.g. in signal processing for military applications, missile warhead guidance systems 
(e.g. uncooled and cooled IR focal plane arrays for heat-seeking missiles, etc).) In both 
Piekarski and T2000, pose / orientation tracking is present, which means that image 
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processing is always being performed in the direction that the user is looking. 
Reference Pham teaches the use of an ATR algorithm that use a focus of attention 
(FOA) algorithm to highlight possible target locations in a full scene, e.g. perform ATR 
based on picking targets out as a human would (Pham 2:37-60). 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the augmented reality and image processing systems 
of Piekarski and T2000 with the ATR FOA algorithms of Pham to allow discrimination of 
targets in the human field of view (including fiducials) in a "fast and efficient manner" 
(Pham 2:42). 

As to claim 14, 

An information presentation apparatus according to claim 9, wherein said virtual image 
generation unit has a function to recognize a target that the user pays attention, by the 
user's operation input to said user operation input unit. 

References T2000 and Piekarski do not explicitly teach the limitations of the 
claim. However, T2000 teaches the use of fiducials for distance and registration 
purposes for indoors and outdoors applications, which means that the camera and 
computer system are performing image processing and automatic target recognition 
(ATR) of the fiducial marks. Further, Piekarski teaches the use of battlefield simulators 
for military purposes, where automatic target recognition is a fundamental of the art 
(e.g. in signal processing for military applications, missile warhead guidance systems 
(radar ones, as in the range of applications for Pham.) In both Piekarski and T2000, 
pose / orientation tracking is present, which means that image processing is always 
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being performed in the direction that the user is looking. Finally, reference T2000 
teaches the use of a two-button input device to fire a simulated weapon with the cross 
hairs being in the middle of the field of view. 

Reference Pham teaches the use of an ATR algorithm that use a focus of 
attention (FOA) algorithm to highlight possible target locations in a full scene, e.g. 
perform ATR based on picking targets out as a human would (Pham 2:37-60). It would 
have been obvious to one having ordinary skill in the art at the time the invention was 
made to combine the augmented reality systems of T200O and Piekarski with the ATR 
FOA algorithm of Pham, since the user would obviously indicate interest in a target by 
shooting it using the device and centering of T2000 and Piekarski, and the ATR 
recognition of Pham would then find the target based on the region the user shot. 
27. Claims 20-21, 24, and 26 are rejected under 35 U.S.C. 103(a) as unpatentable 
over T2000 in view of Piekarski as applied to claim 9 [or other listed claims] above, and 
further in view of Tribes (Sierra™ - Starsiege: Tribes! ® Game) further in view of 
Thomas. 

As to claim 20, 

An information presentation apparatus according to claim 9, wherein said virtual image 
generation unit has a function to generate an annotation indicating a direction of other 
user existing outside a visual range of the user. 

The screenshots below illustrate the markers that the program generates to note 
players on the same and different teams. There are markers for players that are in the 
field of view but not visible if the location is known (e.g. teammates behind walls, behind 
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hills, etc.)(The same concept as employed by the U.S. Army's Blue Force Tracker 
system, for example.) Further, people on the same team can designate targets that will 
be shown to each other using the targeting laser, and a little arrow will show up on 
either side of the screen indicating direction to the target. 




References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
have a range and direction associated with them. Reference Thomas teaches the use 
in section 5 and in Fig. 11 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
sophisticated head-tracking system for pose information. The arrow would prima facie 
follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 1 1 on that same page). In Tribes, 
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as shown in the screenshots above, each team has different color arrows to point to 
teammates and enemies, which convey information on player location. 

It would have been obvious to one having ordinary skill in the art at the time the 
invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas and Tribes, since Piekarski discloses 
tracking arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and target 
annotation in Fig. 7 and Thomas would allow other users / soldiers / players (in the case 
of T2000) to see the targets of interest and guide their attention to them while indicating 
known friendly and unfriendly targets (Tribes) when the targets were outside their field 
of view. 

As to claim 21 , 

An information presentation apparatus according to claim 9, wherein said virtual 
image generation unit has a function to generate an annotation indicating a position of 
other user existing inside a visual range of the user. 

This claim is a substantial duplicate of claim 20; therefore, the rejection for claim 
20 is incorporated herein by reference. The main difference is that "Tribes!" shows on 
the screenshots above that there are arrows pointing to friendly and unfriendly targets in 
the user's field of view (that is, other users are the targets). It would have been obvious 
to one having ordinary skill in the art at the time the invention was made to combine the 
combine the augmented reality of T2000 and the annotation of Piekarski with the arrows 
of Thomas and Tribes, since Piekarski discloses tracking arrows to indicate objects (e.g. 
the helicopter in Figs. 8B and 8C) and target annotation in Fig. 7 and Thomas would 
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allow other users / soldiers / players (in the case of T2000) to see the targets of interest 
and guide their attention to them while indicating known friendly and unfriendly targets 
(Tribes) when the targets were inside their field of view. 
As to claim 24, 

An information processing method according to claim 22, wherein, in a case where the 
attention target exists inside the synthesized image, an annotation indicating additional 
information for the attention target and having an attribute different from that of other 
annotation is generated and synthesized to the synthesized image. 

The screenshots below illustrate the markers that the program generates to note 
players on the same and different teams. There are markers for players that are in the 
field of view but not visible if the location is known (e.g. teammates behind walls, behind 
hills, etc.)(The same concept as employed by the U.S. Army's Blue Force Tracker 
system, for example.) 




References T2000 and Piekarski do not explicitly teach this limitation. Reference 
Piekarski shows in Fig. 8 the use of an arrow pointing at the entity directly in the field of 
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view of the user, and a waypoint is shown off to the left in Fig. 7. All the targets in Fig. 7 
have a range and direction associated with them. Reference Thomas teaches the use 
in section 5 and in Fig. 1 1 of a 3-D pointing arrow. This arrow is taught to mimic the 
behavior of the player virtual avatar, e.g. it points in the direction the user is looking and 
is noticeable by the other players. Thomas suggests is that the user would be able to 
control visibility and location of the pointer. In section 2.2 Thomas disclose the use of a 
sophisticated head-tracking system for pose information. The arrow would prima facie 
follow the user's head orientation. Thomas teaches that it is visible to other users to 
guide them (Thomas section 5.2, pg. 85 and see Fig. 1 1 on that same page). In Tribes, 
as shown in the screenshots above, each team has different color arrows to point to 
teammates and enemies, which convey information on player location, when they are in 
the player's field of view. 

Further, the 3-D arrow disclosed in Thomas is of a significantly different size and 
shape than that used by Tribes to indicate player position. This would meet the recited 
limitation of "having an attribute different than that of other annotation." Also, as noted 
in other rejections, Piekarski includes a forearm keyboard that would prima facie allow 
the user to input annotation that would update the system with regards to the nature of 

the attention target. ._ 

It would have been obvious to one having ordinary skill in the art at the time the 

invention was made to combine the combine the augmented reality of T2000 and the 
annotation of Piekarski with the arrows of Thomas and Tribes, since Piekarski discloses 
tracking arrows to indicate objects (e.g. the helicopter in Figs. 8B and 8C) and target 
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annotation in Fig. 7 and Thomas would allow other users / soldiers / players (in the case 
of T2000) to see the targets of interest and guide their attention to them while indicating 
known friendly and unfriendly targets (Tribes) when the targets were inside their field of 
view. 

As to claim 26, 

An information processing method according to claim 22, wherein an annotation 
indicating a position of other user is generated and synthesized to the synthesized 
image. 

This claim is a substantial duplicate of claim 24; therefore, the rejection for claim 
20 is incorporated herein by reference. The main difference is that "Tribes!" shows on 
the screenshots above that there are arrows pointing to friendly and unfriendly targets in 
the user's field of view (that is, other users are the targets). It would have been obvious 
to one having ordinary skill in the art at the time the invention was made to combine the 
combine the augmented reality of T2000 and the annotation of Piekarski with the arrows 
of Thomas and Tribes, since Piekarski discloses tracking arrows to indicate objects (e.g. 
the helicopter in Figs. 8B and 8C) and target annotation in Fig. 7 and Thomas would 
allow other users / soldiers / players (in the case of T2000) to see the targets of interest 
and guide their attention to them while indicating known friendly and unfriendly targets 
(Tribes) when the targets were inside their field of view. 

Allowable Subject Matter 
28. Claims 19 and 25 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten in independent form including all of the 
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limitations of the base claim and any intervening claims, if rewritten to correct any 112 
issues in the parent claims, and if rewritten so that the claims fall within the four 
categories of patentable / statutory subject matter. 

That is, for claim 19 the references and prior art do not clearly show the 
combination of a changed shape, color, and character string on "an annotation indicates 
a position of other user" as applicant recites. 

For claim 25, the prior art does not show methods for checking that other users 
are viewing the attention target signaled by the first user in an augmented reality 
environment. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Eric V Woods whose telephone number is 703-305- 
0263. The examiner can normally be reached on M-F 7:00-4:30 alternate Fridays off. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Michael Razavi can be reached on 703-305-4713. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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